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ABSTRACT 

A  new  adaptive  dual  control  aolution  la  prevented 
for  the  control  of  a  claaa  of  multi-variable  input- 
output  systems .  Both  rapidly  varying  random  parametera 
and  conatant  but  unknown  par aaatara  are  included.  The 
new  controller  modlfiea  the  cautloua  control  dealgn  by 
numerator  and  denominator  correction  terma.  This  con¬ 
troller  la  ahown  to  depend  upon  aenaitivity  functions 
of  the  expected  future  coat.  A  acalar  example  ia  pre¬ 
vented  to  provide  insight  into  the  properties  of  the 
new  dual  controller.  Monte-Carlo  simulations  are  per¬ 
formed  which  show  improvement  over  the  cautious  con¬ 
troller  and  the  Linear  Feedback  Dual  Controller  of 
U]  and  [2]. 

1.  INTRODUCTION 

Multi-variable  systems  which  are  characterised  by 
uncertain  parametera  with  large  random  variations  are  a 
difficult  challenge  for  moat  control  design  techniques. 
The  assumed  randomess  of  the  parameter  variations 
often  precludes  the  use  of  gain  scheduling  (non  adapt¬ 
ive)  control  design.  Stochastic  adaptive  control 
theory  provides  a  principal  design  approach  for  aysteaa 
of  this  type.  Exact  solution  of  the  stochastic  prob¬ 
lem  with  unknown  parameters  requires  solution  of  the 
Stochastic  Dynamic  Programming  equation  and  this  ia  not 
feasible  for  practical  implementation.  The  solution  ia 
known  to  have  a  dual  effect  [1,2]  Chat  can  be  used  to 
enhance  the  real-time  identification  of  system  paramet¬ 
ers  as  well  as  provide  good  control. 

Many  suboptlmal  dual  solutions  have  been  suggested 
[1,2,5-11].  The  various  approaches  which  have  incor¬ 
porated  this  dual  property  can  be  loosely  divided  into 
two  classes.  Ia  the  first  class  [5-8],  ths  optimal 
control  problem  is  reformulated  to  consist  of  a  one- 
step  ahead  criterion  to  be  minimized,  augmented  by  a 
second  term  which  penalizes  the  cost  for  poor  identifi¬ 
cation.  This  approach  is  attractive  due  to  the  analy¬ 
tical  tractablllty  of  the  solution;  however,  the  solu¬ 
tion  is  based  on  a  one-step  criterion  and  does  not 
fully  exploit  the  dual  property  of  a  multi-step  solu¬ 
tion.  Padilla  and  Cruz  [14]  give  a  dual  control  solu¬ 
tion  for  such  a  plant  by  minimizing  the  control  object¬ 
ive  function  subject  to  an  upper  bound  in  the  total 
estimation  cost.  Their  objective  function  includes  a 
standard  control  objective  function  and  also  a  second 
constraint  term  which  reflects  the  sensitivity  of  ths 
parameters  to  the  state  of  the  system.  Thus  the  solu¬ 
tion  adjusts  itself  to  exercise  better  estimation  for 
such  sensitive  parameters  within  ths  upper  bound.  The 
second  class  [9-11]  utilises  the  stochastic  dynamic 
programming  equation  directly  and  perforas  lineariza¬ 
tion  of  the  future  cost  in  order  to  obtain  a  solution. 
Previous  control  solutions  among  this  sacond  -lass  re¬ 
quire  a  numerical  search  procedure  which  poses  diffi¬ 
culties  for  a  practical  solution  for  on-line  control 
for  multivariable  systems. 

The  linear  feedback  dual  controller  of  [1,2]  is 
•Supported  by  NASA  Axes  Research  Canter  Grant  NAG  2- 
213;  Y.  Bar-Shalon  was  also  supported  from  Air  Force 
Office  of  Scientific  Research  Grant  AFOSR  80-0098. 


based  upon  a  first  order  Taylor  series  expansion  of  the 
expected  future  cost  and  is  called  the  first  order  dual 
(FDD).  It  offers  some  improvement  over  the  non  dual 
cautious  control  based  upon  a  one-stsp  criterion.  The 
results  are  based  upon  a  simulation  model  with  constant 
but  unknown  parameters.  Although  the  dual  control  of¬ 
fers  some  improvement  over  the  cautious  controller  the 
improvement  ia  not  significant  for  most  practical  ap¬ 
plications  where  the  system  contains  constant  parameters 
and  the  objective  is  to  control  in  steady  state  opera¬ 
tion.  However,  for  random  parameter  variations,  dual 
control  can  sometimes  offer  significant  improvement 
over  non-dual  controllars  [5,9].  The  FOD  of  [1,2]  is 
attractive  due  to  its  simplicity  (it  is  comparable  to 
the  cautloua  control  design  in  algorithm  complexity  su'd 
does  not  require  numerical  search).  The  objective  of 
the  present  study  is  to  evaluate  the  cautious  control¬ 
ler  and  the  FOD  for  large  random  parameter  variations 
modeled  as  a  random  walk.  Monte-Carlo  simulations  are 
performed  and  conditions  quantified  under  which  the 
dual  controller  offers  significant  improvement  over  a 
non-dual  cautious  controller. 

The  FOD,  although  offering  a  reduction  in  the  aver¬ 
age  cost,  is  found  to  be  unacceptable  In  many  cases. 

This  is  attributed  to  the  sensitivity  of  the  expected 
future  coat  whenever  the  system  is  characterized  by 
limited  controllability.  A  second  order  expansion  of 
ths  linearization  procedure  of  [1,2]  is  presented  to 
account  for  this  sensitivity.  This  new  second  order 
dual  controller  (SOD)  inherently  includes  a  robustness 
property  in  that  the  controller  accounts  for  sensiti¬ 
vity  of  the  expected  future  cost  due  to  parameter  esti¬ 
mates  and  their  uncertainty.  Simulations  are  presented 
which  show  the  Improvement  of  the  SOD  over  the  cautious 
controllar  and  the  FOD.  This  SOD  uses  a  Newton  type 
search  procedure  and  is  developed  for  multi-variable 
systems.  One  of  the  main  advantages  of  the  SOD  pre¬ 
sented  herein  is  that  it  modifies  the  cautious  control¬ 
ler  with  a  numerator  "probing"  term  and  a  denominator 
correction  term.  Although  the  SOD  is  still  considered 
too  complex  for  practical  implementation,  the  structure 
of  the  coatrol  solution  is  in  s  form  which  permits 
practical  design  changes  to  ths  cautious  controller  to 
include  the  dual  properties. 

Section  2  gives  ths  problem  formulation.  Ths  ap¬ 
proximate  dual  controller  for  the  multi-variable  input- 
output  system  is  developed  in  Section  3.  Section  4 
analyzes  this  dual  controller  for  a  scalar  example  with 
one  unknown  parameter.  Section  5  concludes  the  paper. 

2.  PROBLEM  FORMULATION 

Ths  multivariable  system  under  investigation  is 

x(k+l)  -  c(k>  +  BCk)  u(k)  (2.1) 

where  c(k)  la  an  unknown  vector  and  B(k)  is  a  matrix  of 
unknown  parameters.  The  unknown  s laments  of  c(k)  and 
B(k)  are  denoted  as  8(k)  with  covariance  matrix  P(k) . 
These  are  represented  by  s  discrete  random  model 

8(k+l)  -  A0(k)  +  v(k)  (2.2) 

yv  s,  g4 
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The  measurement  equation  la 

yCk)  -  x(k)  +  w(k)  (2.4) 

where 

E(w(k) )  -  0  end  E(w(k)w'(J))  -  H  <$. 

3  (2.5) 

E(w(k)v,(j))  -  0 

and  x(k),  yCk)  being  n  dimensional  vectors.  The  control 
criterion  to  be  minimized  Is  the  expected  value  of  the 
cost  from  step  0  to  N 

N 

J(0)  -  E{C(0)  }  -  E  {  Z  x'(k)Qx(k)+  u‘  (k-l)Ru(k-l)  } 

k'X  (2.6) 

where  N  •  2  for  the  two  step  ahead  criterion. 

3.  APPROXIMATE  DUAL  CONTROLLER  TOR  TWO  STEP  CRITERION 

The  minimization  of  (2.6)  with  respect  to  u(0)  and 
u(l)  subject  to  (2.1)  -  (2.5)  is  obtained  from  the 
Stochastic  Dynamic  Programming  equation  [12,131 

J*(k)  -  min  E{c(k)+J*(k+1)  |Yk]  k-N-1, . . . ,1,0  (3.1) 

u(k) 

where  J*(k)  Is  the  "cost-to-go"  from  k  to  N  and  Yk  is 
the  cumulated  Information  at  time  k  when  the  control 
u(k)  Is  to  be  determined.  For  N  -  1,  (3.1)  Is 

J*(0)  -  mln  E{x* (l)Qx(l)+ti' (0) Ru(0)+J*(l)  Iy0}  (3.2) 

u(0) 

where  J*( 1)  Is  the  optimal  cost  at  the  last  step  and  la 
obtained  by  minimization  of  J(N-l)  for  N  -  2.  Assuming 
diagonal  Q  -  diag(q^)  this  results  In  [1,2] 

J*(l)  -  c'(l)Qc(l)  +  Z  q,pf(l)  (3.3) 

*-l  lc 

-  [c*  (l)QB(l)  +  q^P^U)]  [B'(1)QB(1)  + 

+  ”  q»P»(l)  +  R]”1,  [B,(l)Qc(l)+  E  q/  (1)1 

*■1  c  *  *.  it!c 

and  ^ 

u*(l)  -  -[B’d)QB(l)  +  Z  q^-g(l)  +  R]'1[B'(l)Qc(l) 

"  t 

+  Z  q.P,  (1)  1  (3.4) 

*-l  L  Bc 


P^d)  pj,(l) 
Pf  (1)  p5(1) 


P(l)  Is  the  expected  velue  of  (8(1))*  for  time 
step  2  given  measurement  y(l)  at  time  step  1.  The  In¬ 
dex  l  la  used  to  represent  the  row  number  In  (2.1)  ad 
P*(l)  Is  the  associated  parameter  covariance. 

The  parameter  estimates  8(1)  and  covariances  P(l) 
are  obtained  from  the  Kalman  filter.  Since  W  is  diag¬ 
onal  one  can  decouple  the  estimation.  Than 


8^(1)  -  A8t(0)+AKf(l)  V^(l) 

K^l)  -  Pt(0)H,(l)[H(l)P£(0)H,(l)+Wi]' 
P*(l)  -  tl(0)  -  K^(1)H(1)P^(0) 


P*(l)  -  AP*(1)A’  +  V 


V^d)  -  y^d)  -  H(l)8t(0) 
H(l)  -  [1  uT(0)) 


9*(1)  -  (C^d)  B^(l)  ]T  ,  *»1,2,  ...n  row  of  B  (3.12) 


As  discussed  In  [1]  and  [2]  J  Q)  la  a  nonlinear 
function  of  the  parameter  estimates  8(1).  and  covariances 
P(l)  and  thus  a  linearization  was  performed.  In  [1]  a 
scalar  formulation  was  presented  and  a  first  order  lin¬ 
earization  was  performed  about  the  nominal  parameter 
estimate  squared  (6(0)) 2  and  nominal  eovarlance  F(l) . 
Also  In  [1,2]  the  vector  case  was  presented  and  linear¬ 
ization  to  first  order  performed.  To  more  accurately 
account  for  the  dual  effect  e ^second  order  Taylor  Serlea 
expansion  is  presented  about  8(0)  and_a  first  order  ex¬ 
pansion  about  the  nominal  covariance  P(l).  In  addition 
(as  will  be  presented  subsequently)  the  c overlance  P(l) 
will  Include  a  linearization  to  second  order  In  u(0) . 

In  [1,2],  P(l)  was  linearized  to  flrat  order.  It  is 
believed  that  linearizations  to  second  ordar  are  neces¬ 
sary  to  better  account  for  the  nonlinearity  In  P(l)  and 
8(1)  of  (3.3)  and  In  u(0)  of  (3.7)  mad  (3.8).  In  addi¬ 
tion  a  nonlinear  Newton  algorithm  la  used  In  the  second 
order  approximation. 

Linearization  of  (3.3)  about  the  nominal  9(1) 

■  A0(O)  and  P(l)  using  the  nominal  u(0)  results  In 

J*(l)  -  J*[l,  6(0),  P(l)]  +  [8(1)  -  A6(0) ] 

36(1) 

+  |[8(1)  -  A8(0) ] *  [8(1)  -  A8(0) ] 

*  a^d) 


+  Z  Z  Z  [pf  (1)  -  ff  ( 

*■1  1-1  J-l  3P7  .(1)  1,3 


where  the  superscript  l  represents  the  covariance  matrix 
associated  with  the  *th  row  of  parameters  and  Pt  , (1) 
is  the  i-j  th  element  of  tha  covariance  matrix  1 
P(l),  n  being  the  number  of  unknown  parameters. 

Using  (3.6)  tha  expected  value  of  (3.13)  Is 

E[J*(1)  |Y°]  -  J*[l,  8(0),  f(l)] 

+  j  tr[  K(l)  E{v(i)v*(l)  |yo}K.(1)  j 

*  382(1) 

+  S  “  ’  ■  [pf  .(1)  -  ?[  .(1)]  (3.14) 

*-l  1-1  J-l  3pf  .(1)  1,3  1,3 

Using  (3.7),  (3.8)  and  tha  innovation  covariance 
E{v£(1)  v£(1)|y°}  -  H(l)P*(0)H’ll)  +Vt  (3.15) 

(3.14)  can  be  written  as 


E[J*(1)|Y°]  -  J*[l,  8(0),  P(l)] 


♦  r  “  I 

*-l  1-1  J-l  (  *  38^1)  38j(l)  I— 1,3 

-vf 7]  +  [pf  j(D-pf  j(i)]j 

3P*J(1)  ,3  °  ) 


38^1)  38j  (1)  I 


CU-AP^WU’ 


funct Ion 


Tha  expected  future  cost  (3.16).  is 
Ion  of  the  predicted  covariance  Pt 


shown  to  be  a 
,  (1)  with  a 


multiplier  given  by  the  sensitivity 

laro) _ 


38^1)36^1) 


Since  the  covariance 


dual  effect.  It  should  be  noted  that  tha  Importance  of 
the  dual  effact  depends  upon  tha  sensitivity  of  the  ex¬ 
pected  future  cost  with  respect  to  both  the  covariance 
and  parameter  estimate. 

The  optimal  control  u(0)  can  be  computed  by  mini¬ 
mization  of  (3.2)  using  (3.16).  Since  Pr  ,(1)  Is  non- 

linear  In  u(0)  a  numerical  search  procadure  is  required. 
This  la  accomplished  using  a  second  order  linearisation 


2 


\  v.  *. 


la  u<0). 

Thu*  (3.8)  la  linearised  to  second  order  about  the 
control  u1(0)>  which  la  In  the  vicinity  of  the  optimal 
control. 

,  .  apf  ,u>  x 

pMa)  ■  ♦sJr-  i,'rw>-“ <0)l 


.  .  32pf  .(1)  , 

+  i(u(0)-u(0))’  - -  (u(O)-u  (0)  ]  (3.17) 

3  u2(0)  ul(0) 

The  expected- future  coat  a*  given  by  (3.16)  and 
(3.17)  la  quadratic  In  u(0)  and  thua  e  cloaed  fora 
solution  u*(0)  la  obtained  by  ainialsation  of  (3.2). 

The  optimal  dual  control  u*(0)  can  now  be  computed 
from  (3.2)  uelng  (3.16)  and  (3.17).  It  la  obtained  by 
aolvlng  (3. 18) 

-j^Etx'UX)  x(l)  +  u*(0)Ru(0)  +  J*(1)|Y°}  -  0 

The  optimal  u*(0)  la  thus 

-  ^  n  t  . 

u*(0)  -  -(B'(O)QB(O)  +  t  (q.PZXO)  +  F.)  +  R]-i 
f- 1  *  B  L 

(B'(O)Qc(O)  +  Z^  (q^Pjc(0)  +  f^)l  (3.19) 

where  the  matrix  F^  and  the  vector  f^  are 

r  .  “  *  1  /  .  .  1.  3-jAij\ 

1  1-1  J-l  2  \3P^  *(l)  2  38^(1)  3^(1)  / 


»1.1(1) 

3u(0) 


■  ■  1 

z  z  j 

1-1 1-1  * 


uI(O),0(O).F(l) 


1  3 

2^ 


a  »*!»)  i  \ 

3u(0)  3u(0)  “  WJ  uI(0)ie(0),p( 


the  more  Important  will  be  the  dual  effect. 

The  resulting  dual  controller  (3.19)  exhibits  a 
robustness  property  with  respect  to  parameter  variations 
and  uncertainty  of  the  future  cost  by  including  a  ten 
which  appears  In  the  denominator  of  the  dual  controller. 
In  addition,  a  probing  ten  appears  In  the  numerator. 

A.  SCALAR  EXAMPLE  WITH  ONE  UNKNOWN  PABAtgTBk 

To  further  understand  the  dual  control  solution  a 
scalar  example  with  one  unknown  parameter  b  is  presented. 
The  approxlMte  dual  control  solution  for  this  scalar 
case  using  Q  •  1,AX  ■  0,  Is  given  by  (3.19)  -  (3.2J) 
with  P,  (1)  and  8(0)  being  replaced  by  Pb(l)  and  b(0) 
respectively. 

The  partials  required  In  the  control  law  ere 


|b(0)  ,Pb(l) 


2*2 

c  b  (0) 

(b2(0)+Fb(i))2 


a2i*m  2  Fh(l)-3b2(0)  . 

-  -2cr.  (1)  -§ - ,  t4-2) 

3b(l)  3b(l)  ;(0)  ?  (1)  (b2(0)+Pb(l))3 

b 

».  (1)  I  2P?(0)H  ux(0)a2 

b _  I  _  ..  b _ t  l 


32pb(D 

3u(0)3u(0) 


(Pb(0)uI  (O)-HJ)2 


,  W-3P,  (0)u  (0)  , 

•  -  -2P?(0)W  - £ - - -  » 

"  <0)  ^(OJu1  (O)-H#)3 


Initially  the  nominal  value  of  u(0)  Is  computed 
from  (3.19)  with  F£  and  f^>  equal  to  aero.  Then  a  grad¬ 
ient  search  Is  performed  until  In  the  vicinity  of  the 
optimal  u*(0) .  Then  (3.19)  -  (3.21)  are  used  until 
convergence  Is  achieved.  This  Iteration  procedure  is 
essentially  Newton's  method  for  minimization  of  a  non¬ 
linear  function.  The  gradient  search  Is  used  because 
the  stochastic  cost  In  (3.2)  being  minimised  Is  a  high 
order  nonlinear  equation  and  the  gradient  procedure  Is 
used  until  ur(0)  Is  In  the  vicinity  of  the  minimum 
before  switching  to  the  Newton  method.  The  nominal 
covariance  ?£(1)  Is  computed  from((3.7)  -  (3.11))  with 
u(0)  •  u(0) .  The  sensitivity  (partials)  In  (3.20)  and 
(3.21)  of  the  cost  J*(l)  are  computed  from  partial 
derivatives  of  J*(l)  (3.3)  and  P*(l)  (3.7)  -  (3.9) 
evaluated  at  the  nominal.  The  partials  of  the  covari¬ 
ance  are  evaluated  at  u*(0)  which  is  evaluated  at  the 
previous  Iteration  I. 

The  approximate  two-step  ahead  dual  control  of 
(3.19)  -  (3.21)  can  be  Interpreted  aa  a  modification  to 
the  cautious  controller  by  the  terms  Ff  and  These 
terms  depend  upon  the  sensitivity  of  the  future  nominal 
cost  J*(l)  with  respect  to  the  parameters  §J(1)  0v(l) 
for  all  l,j  and  their  covariance  Pr  ,(1)  for  ascA  row 
t  of  parameters.  Whenever  these  sedaitlvities  are 
large  the  terms  F/  and  f/  will  be  significant  (that  la 
the  dual  effect  will  be  Important).  Thus  the  sensiti¬ 
vities  taka  Into  account  In  the  control  solution  tha 
sensitivity  of  the  nominal  future  cost  due  to  parameter 
variation  and  uncertainty.  The  larger  this  sensitivity 


where  the  nominal  u(0)  and  F.  (1)  are 
«  b 


b  (0)+?b(0) 
e2PK(0)V 

p  (1)  - - * - +  V  (4.6) 

B  P  (0)i»  (0)+W 

b  « 

The  parameter  estimate  b(0)  and  Pb(D)  are  computed 
using  data  up  to  k  -  0  (l.e.  y(0)). 

The  expected  future  coat  based  upon  tha  lineari¬ 
sation  of  (3.16)  is  , 

t(j’u) It”)  -  c2  -  t!  4^ - 

b*(0)+FbU)  1  3b  1 1) 

-  a2Pb(0)-V)  +  (Pba)-FbU))  (*.7) 

b 

4.1  Evaluation  of  the  Cautious  Controller 

The  performance  of  the  cautious  controller  can  be 
evaluated  using  (3.2)  with  u(0)  evaluated  at  the 
nominal 

J(0)  -  [E{x2C1)  |t°)  +  E{J*(1)  l*°>]uC0).n(D)  (4.8) 

Tha  first  term  In  (4.8)  represents  tha  expected  cost  at 
k  -  1  and  the  second  term  In  (4.8)  represents  the  ex¬ 
pected  future  cost  at  k  -  2  using  tha  cautious  control 
at  k  •  2  (l.e.  u(l))  and  using  tha  cautious  control  at 
k  -  1  (l.e.  u(0)-u(0)).  (4.8)  is  evaluated  using 

data  Y°. 

Using  (4.1)  -  (4.7),  (4.8)  becomes. 


J(0)  -  c  - 


b*(0)+Pb(D) 


,  2  2 
+  c  -  e 


b‘(0)+Pb(l) 


.  1  32J*(1) 

+  l  *2 


a2Pb(0)u2(0) 


3b  (1)  Pb(0)u  (fl)4W 


*  a  •  >a  '  *JW  *  --  •  *  -J* 


3 


The  last  ttn  In  (4.7)  la  saro  since  P.  (1)  evalu¬ 
ated  at  the  nominal  control  (l.e.  cautious  control) 
equals  5^(1) .  The  flrat  two  ter ms  In  (4.9)  represent 
the  average  cost  at  step  k  «  1  and  the  last  three  terms 
represent  the  expected  future  cost  at  k  •  2  using  the 
cautious  control. 

A  simple  example  can  be  used  with  (4.9)  to  demon¬ 
strate  when  the  cautious  control  la  expectad  to  behave 
poorly. 

Assume  a  scalar  example  with  one  unknown  b  para¬ 
meter  and  let 


The  expected  cost  at  k  •  1  and  k  •  2  la  computed 
from  the  nominal,  3(0),  P,  (1)  and  3rJ*(l)  which  yields 


3(0)  -  -.1 


J(0)  =  c2  +  c2  ,  c  -  1  (4.12) 

Thus  the  cautious  control  applied  at  k  -  0  results 
In  no  reduction  In  the  cost  at  k  *  1  due  to  large  un¬ 
certainty  P(l)  and  also  no  reduction  In  the  future  ex¬ 
pected  cost  since  3(0)  Is  small  and  no  Improvement  In 
parameter  accuracy  occurs  at  step  k  «  1. 

4.2  Evaluation  of  the  Dual  Controller 

The  dual  controller  of  (3.19)  -  (3.21),  (4.1)  - 
(4.6)  can  be  evaluated  by  computing  the  average  coat  of 
(4.8)  using  the  covariance 
«2Pb«»W 

P.C1) - ^5— -  +  V  (4.13) 

Pb(0)u^(0)+W 

The  expected  future  cost  (4.7)  reduces  to 


[1)  and  yj*(l) 

2  .  3^(1) 

,  -  - 

3bZ(l) 


e{j\i)|y°}| 


b“(0)+Pb(l) 


,  1  aV(i)  «2pbWu*2<°) 

2  3b2 (1)  Pb(0)u*2(0)+W 


.  jjja> 


/•rbco)' 

\  P.  (0)u* 


u*2^) 


a2P2(0)32(0) 
Pb(0) J2(0)+«  / 


and  the  total  expected  cost  at  k  •  1  and  k 
(4.8)  Is 

J*(0)  -  E{x2(1)|y°}|  +  E{J*(1)|Y°}| 


2  using 


e{x2(1)|Y0}  -  c2  +  2b(0)u*(0)c  + 

u*(0) 

+  (^(0)  +  Pb(0))u*2(0)  (4.16) 

Examination  of  (4.14)  shows  that  the  dual  control 
can  reduce  the  expected  future  cost  over  the  cautious 
control  since  the  last  two  expressions  in  (4.14)  can  be 
negative  If  u"*(0)  >  u2(0).  Thus  the  dual  property 
can  have  a  desirable  effect  on  the  future  cost. 

The  cost  J  (0)  Is  computed  using  the  scalar  exam¬ 
ple  previously  discussed  for  the  cautious  controller. 

A  search  procedure  la  used  on  (4.13)  using  (4.14)  and 
(4.16)  with  the  parameter  values  from  (4.10),  and  u*(0) 
Is  Iterated  until  in  the  vicinity  of  the  minimum 
yielding 


irgj.  .  -  -0073  ,  jlfr-W  -  -3.47 , 

uto)— .1  3b  (1)  3(0)"-.  1 


3Pb(l)  32P.(1) 

■5355“  T  ■  -382  •  ~r — 

mZ(0)—.6  W) 

F.  -  .87  ,  f.  -  .85 


t  “  +1.0  , 

u‘(0)— .6 


The  above  sensitivities  (4.17)  were  evaluated  in 
the  vicinity  of  the  optimal  u*(0)  -  -.6  and  P.  (D-.278. 
The  dual  control  u  (0)  using  ur(0)-  -.6,  e«l  Is 


b(0)  c  4-  ■  85 
£(0)  +  Pb(0)  +  .87 


•  -.62 


The  corresponding  future  expected  cost  using  (4.14) 
and  (4.17)  is 


E{J  (1)  |y°}| 


1  ij&u 

2  3b2  (1) 


p^UOu^oo) 

Pb(C)u*2(0)-W 


=  .442  c  ,  c  -  1  (4.19) 

The  result  of  this  example  shows  that  the  dual 
control  of  (4.18)  reduces  the  expected  future  cost  to 
44Z  of  the  original  c2  with  no  control.  The  cautious 
control  resulted  In  no  reduction  of  the  future  cost. 

The  tarns  responsible  for  the  Improvement  with  dual  con¬ 
trol  are  tha  second  order  sensitivities  32PU)  and 

4£Ul  . 

3d  (1) 

Tha  dual  control  of  (4.18)  differs  from  the  cau¬ 
tious  control  (4.11)  by  the  terms  -  .87  In  the  denom¬ 
inator  and  f£  ■  .83  In  the  numerator.  The  denominator 
term  In  effect  provides  more  "caution"  whereas  the 
numerator  term  Is  an  additive  probing  effect.  The  term 
P 1  provides  a  "robustness"  property  in  that  the  sensi¬ 
tivity  of  the  future  cost  to  parameter  uncertainties  as 
they  appear  In  tha  controller  (l.e.  ^(0))  are  minimised. 
Thus  a  new  interpretation  of  the  dual  control  Is  that  It 
contains  robustness  and  learning  Cvia  probing) .  These 
concepts  are  applicable  to  the  multivariable  dual  con¬ 
troller  In  (3.19)  -  (3.21). 

5.  SIMULATION  RESULTS 

Performance  was  evaluated  from  100  Mqnte  Carlo 
runs  for  the  following  controllers  where  b(0)  was  set 
to  b(0)  with  covariance  Pb(0) :  1)  Cautious  Controller 

2)  FDD  3)  SOD 

The  above  algorithms  were  tested  for  two  cases: 

a)  Time  varying  case,  b(0)  ■  .05,  Pb(0)  •  1.0  , 

V  -  .1,  c  -  1.0,  W  -  .01  and  M  -  .1,  a  -  0.9 

b)  Constant  case,  with  b(0)  ■  .05,  PbC0)  -  1.0, 

V  -  0,  c  -  1.0,  W  -  .01  and  V  -  .1,  a  -  1.0 
Example  a 

Table  1  summarises  tha  results  of  the  simulation 
runs.  All  three  algorithms  were  tested  on  this  example 
for  tiro  different  levels  of  measurement  noise  covariance, 
W  •  .01  and  W  •  .1.  100  Monte  Carlo  tuna  were  performed 

each  of  40  time  steps.  For  each  run,  an  average  cost 
was  computed  over  40  time  steps  and  than  tha  averages 
over  100  runs  are  tabulated  In  Table  1  and  Table  2. 

Tha  tables  clearly  Indicate  that  tha  SOD  yields  the 
least  cost.  The  dual  effect  shows  a  larger  Improvement 
for  larger  measurement  noise  (l.e.  W  -  .1).  tun  numbers 
7  and  14  of  the  100  Monte  Carlo  runs  were  selected  for 
plotting.  Tha  cost  and  parameter  value  are  plotted  In 
Figures  1  through  4.  It  Is  evident  that  the  second 
order  dual  Improves  upon  the  other  two  on  the  average. 
Example  b 

In  this  case  the  true  parameter  was  close  to  sero 
(l.e.,  b(0)  ■  .05)  but  constmt.  Table  2  sumsarlxea 
the  result.  The  average  cost  obtained  by  the  SOD  Is 


■uch  lower  than  tha  other  two.  ■  Tha  SOD  always  axhibltad 
excellent  convergence  whereas  tha  othar  controllers  per¬ 
formed  poorly.  In  addition  tha  new  controller  consist¬ 
ently  avoided  turn  off  and  burst  [S ] .  This  was  an  im¬ 
portant  ccasoo  feature  in  all  the  Monte  Carlo  runs. 

Runs  26  and  80  are  plotted  in  Figures  5  end  6  respect¬ 
ively,  as  typical  examples. 

The  simulation  study  has  shown  that  the  new  dual 
controller  improves  upon  the  cost  on  the  average.  The 
magnitude  of  the  improvement  on  the  average  appear a  to 
be  relatively  small  for  tha  noise  levels  used.  However, 
the  real  advantage  of  the  new  dual  controller  is  the 
improvement  in  those  lnatances  where  the  cautloua  con¬ 
troller  and  the  FOD  (1,2)  yields  unacceptable  results. 
Although  the  FOD  [1,2]  shows  improvement  over  the  caut¬ 
ious  controller,  it  hes  been  found  to  be  unacceptable 
at  many  time  points. 

6.  CONCLUSION 

A  new  adaptive  dual  control  solution  based  upon  the 
sensitivity  functions  of  the  expected  future  cost  has 
been  presented.  This  controller  (SOD)  takas  into  ac¬ 
count  the  dual  effect  better  by  performing  the  second 
order  Taylor  series  expansion  of  the  expected  future 
cost.  The  form  of  this  controller  is  a  modification  of 
the  one  step  cautious  controller.  The  FOD  of  [1,2]  did 
not  have  the  denominator  correction  term  like  the  pre¬ 
sent  one.  This  adds  stability  to  the  osw  control  de¬ 
sign.  Simulation  results  of  a  scalar  model  have  shown 
the  Improvement  obtained  using  the  new  dual  algorithm. 
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Fig.  1.  Tine  history  of  cost  comparing 
the  SOD,  FDD,  and  the  cautious 
controller  (Time  varying  parameter 
case:  Run  No.  7  from  100 
Monte  Carlo  kins) . 
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Time  history  of  cost  comparing 
the  SOD,  F0D,  and  the  cautious 
controller  (Time  varying  parameter 
case:  Run  No.  14  from  100 

Monte  Carlo  Runs) 

Fig.  5.  Time  history  of  cost  comparing 
the  SOD,  FOD,  and  tbs  cautious 
controller  (Constant  parameter 
case:  km  No.  26  from  100 

Moots  Carlo  kins). 
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Time  history  of  parameter 
for  Run  No.  14  from  100 
Monte  Carlo  Runs 
(Time  Varying  Casa) . 
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Fig.  6.  Time  history  of  cost  comparing 
the  SOD,  POD,  and  the  cautious 
controller  (Constant  parameter 
caae:  Run  No.  BO  from  100 
Monte  Carlo  Runs) . 


Average  Cost 

Cautious 

First  Order 
Dual 

Second  Order 
Dual 

.475 

.469 

.438 

.623 

.608 

.514 

Measurement 

Noise  Covariance 
W 


Cautious 

First  Order 
Dual 

Second  Order 
Dual 

.109 

.087 

.069 

.359 

.250 

.142 

Table  1.  Average  Coot  for  the  three  controllers  on  the 
time  varying  parameter  model  (b(O)-.OS, 
Pb(0)-1,  V-.l,  c-1) 


Table  2.  Average  Cost  for  three  controllers  on  the 

Constant  Parameter  Model  (b(0)-.05,  P.  (0)-l, 
V-0,  c-1) 
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